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Preface 


The  Department  of  Defense  (DoD)  relies  on  risk  management  analysis  when  acquiring  large 
defense  acquisition  programs.  Risk  management  helps  decisionmakers  ensure  that  objectives 
related  to  cost,  schedule,  and  performance  are  met  according  to  program  goals.  To  that  end, 
a  team  of  RAND  researchers  created  a  Microsoft  Excel-based  tool  (the  Assessor  Tool)  to  help 
DoD  acquisition  specialists  identify  system  integration  risk  areas  at  any  point  in  the  acquisi¬ 
tion  process.  This  document  offers  a  users’  manual  for  the  current  integration  risk  application 
of  the  Assessor  Tool  (Version  1.0)  and  instructions  for  how  to  adapt  the  Assessor  Tool  for  dif¬ 
ferent  applications.  A  complementary  report  describing  the  methodology  behind  the  tool  and 
its  applications  is  available  as  RR-262-OSD,  A  Risk  Assessment  Methodology  and  Excel  Tool  for 
Acquisition  Programs  (Fleishman-Mayer,  Arena,  and  McMahon,  2013). 

This  work  should  be  of  interest  to  those  readers  interested  in  risk  assessment  of  major 
defense  programs.  The  document  does  not  assume  an  understanding  of  the  DoD  acquisition 
system.  This  research  was  conducted  within  the  Acquisition  and  Technology  Policy  Center 
of  the  RAND  National  Defense  Research  Institute,  a  federally  funded  research  and  develop¬ 
ment  center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the  Unified 
Combatant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agencies,  and  the  defense 
Intelligence  Community. 

For  more  information  on  the  Acquisition  and  Technology  Policy  Center,  see  http://www. 
rand.org/nsrd/ndri/centers/atp.html  or  contact  the  director  (contact  information  is  provided 
on  the  web  page). 
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Material  Development  Decision 
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CHAPTER  ONE 


Introduction 


On  May  22,  2009,  President  Obama  signed  into  law  the  Weapon  Systems  Acquisition  Reform 
Act  (WSARA)  to  improve  program  costs  and  schedules  associated  with  the  delivery  of  major 
weapon  systems.  Some  of  the  oversight  changes  called  for  by  WSARA  depend  on  a  program 
team’s  ability  to  measure  and  manage  the  various  risks  associated  with  system  integration  (SI). 
Because  SI  may  be  influenced  by  all  elements  of  the  acquisition  process,  there  exists  a  wide 
range  of  sources  for  SI  risk.  At  any  point,  problems  with  hardware  or  software,  design  matu¬ 
rity,  timely  funding,  test  plan  execution  and  personnel,  facilities,  and  supplier  capabilities  can 
negatively  affect  program  cost,  timelines,  and  performance  goals.  Historically,  integration  risks 
at  various  phases  of  the  acquisition  process  have  contributed  in  part  to  program  delays  and  cost 
overruns.  In  response,  the  U.S.  Department  of  Defense  (DoD)  has  worked  toward  improving 
defense  program  management  overall  through  program  and  contractor-level  risk  management 
practices  (DoD,  2006). 

Large  defense  programs  can  have  many  technical,  legal,  and  political  consequences.  Thus, 
there  are  many  stakeholders  across  DoD  who  need  to  identify  the  risks  associated  with  DoD’s 
overall  weapons  programs,  as  well  as  the  individual  technology  projects  within  a  program.  To 
date,  personnel  from  the  Office  of  the  Secretary  of  Defense  (OSD)  who  have  been  more  gener¬ 
ally  involved  with  weapon  systems  acquisition — but  not  necessarily  involved  with  individual 
programs — have  had  no  access  to  an  OSD-level  systematic  method  of  determining  a  program’s 
ability  to  meet  its  goals,  or  to  monitor  the  success  of  the  defense  sector’s  compliance  with 
WSARA  over  the  acquisition  lifecycle.  The  methods  currently  available  to  OSD  personnel  are 
too  technically  focused  and  are  relevant  only  to  personnel  who  have  detailed  knowledge  at  the 
individual  program  level. 

The  Excel  information-based  risk  tool  (referred  to  as  the  “Assessor  Tool,”  or  “tool”  for 
short,  for  the  remainder  of  the  document)  described  herein  (Version  1.0)  is  designed  to  assist 
the  DoD  acquisition  community  in  assessing  weapon  SI  risk  in  accordance  with  WSARA.  A 
complementary  report  describing  the  methodology  behind  the  tool  and  its  applications  is  avail¬ 
able  as  RR-262-OSD,  A  Risk  Assessment  Methodology  and  Excel  Tool  for  Acquisition  Programs 
(Fleishman-Mayer,  Arena,  and  McMahon,  2013).  The  package  offers  an  OSD-level  approach 
to  the  evaluation  and  measurement  of  SI  risk.  That  is,  it  is  meant  for  assessors,  such  as  OSD 
personnel,  who  may  not  be  especially  familiar  with  the  specific  program  under  evaluation  but 
still  may  need  to  make  judgments  about  the  program’s  risk.  The  tool  is  a  custom-designed 
software  package  in  Excel  that  allows  for  easy  accessibility  of  an  OSD-level  audience.  Other 
systems  engineering  (SE)  risk  management  software  tools,  such  as  a  COTS  (commercial  off- 
the-shelf)  SE  tool  (e.g.,  Lebron,  Rossi,  and  Foor,  2000),  may  not  be  appropriate  or  easily  avail- 
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able  for  this  audience.  While  potentially  not  unique,  the  tool  is  tailored  specifically  to  OSD 
personnel,  allowing  for  its  ease  of  use. 

The  Assessor  Tool,  developed  by  RAND  researchers,  is  based  on  a  tractable  and  com¬ 
prehensive  set  of  questions  that  can  help  evaluate  integration  risk  at  each  point  in  the  acquisi¬ 
tion  process.  More  specifically,  the  tool  enables  users  to  see  how  well  integration  risk  is  being 
managed  by  providing  a  standards-based  valuation  of  integration  issues  that  can  lead  to  cost 
growth,  schedule  growth,  and  program  performance  shortfalls.  These  standards  are  based  on 
the  existence  and  completeness  of  DoD  artifacts  and  checklists  that  would  be  readily  avail¬ 
able  to  an  assessor  at  the  OSD  level.  As  requested  by  the  OSD  sponsor,  we  developed  the 
tool  and  its  methodology  to  help  OSD-level  acquisition  professionals  address  these  potential 
risks  to  major  programs;  early  identification  and  reconciliation  of  SI  issues  as  mandated  by 
WSARA  can  reduce  the  likelihood  and  magnitude  of  the  complications  that  frequently  affect 
major  weapons  acquisition  programs  (Conrow,  1995).  While  we  describe  the  Assessor  Tool  in 
terms  of  its  appropriateness  for  major  weapon  systems  acquisitions  analysis,  it  should  be  noted 
that  it  is  also  generalizable  to  an  entire  set  of  OSD-level  information-based  risk  assessment 
applications. 


CHAPTER  TWO 

User  Manual  for  Assessor  Tool 


The  Assessor  Tool  (Version  1.0)  described  in  this  document  can  be  used  in  its  existing  applica¬ 
tion  for  assessing  a  weapon  systems  acquisition  program’s  integration  risk,  as  well  as  adapted  for 
other  programs  and  compliance  risk  assessments.  This  chapter  provides  step-by-step  instruc¬ 
tions  to  perform  both  of  these  functions.  It  assumes  that  users  have  read  the  main  report 
(Fleishman-Mayer,  Arena,  and  McMahon,  2013)  and  therefore  does  not  provide  detailed  defi¬ 
nitions  of  terms  such  as  “secondary  questions”  or  “relevant  artifacts.” 


Instructions  to  Use  the  Existing  Application  of  the  Assessor  Tool  (Weapon 
Systems  Acquisition  Integration  Risk) 

This  section  contains  step-by-step  instructions  for  a  user  of  the  existing  integration  risk  Asses¬ 
sor  Tool.  The  instructions  will  lead  the  user  through  an  exercise  to  assess  a  specific  weapon 
systems  acquisition  program  at  a  specific  program  phase  (ASR,  PDR,  etc.).  A  user  can  use  these 
instructions  to  answer  the  existing  phase-specific  and  global  questions  as  to  their  completeness 
and  importance  and  can  add  questions  tailored  to  the  program  under  assessment. 

1.  Open  the  Excel  file,  enabling  macros  if  possible. 

2.  The  Excel  file  should  open  to  the  overview  tab  (see  Figure  2.1).  If  not,  find  this  tab  at 
the  bottom  of  the  window  and  click  on  it. 

3.  At  the  top  of  the  page,  fill  in  the  program  name  in  the  area  highlighted  in  gray. 

4.  Click  the  button  for  the  appropriate  program  phase  (see  Table  2.1)  (ASR,  PDR,  etc.). 

5.  At  the  top  of  the  page,  fill  in  the  assessor  name  and  date. 

6.  For  each  program  phase  question  (highlighted  in  blue),  read  the  question.  If  it  is  a 
primary  question,  choose  its  level  of  importance  on  a  scale  from  1  (Little  Importance) 
to  5  (Extremely  Important).  Next,  for  all  primary  and  secondary  questions,  choose 
its  appropriate  “Assessment”:  Addressed,  Partially  Addressed,  Not  Addressed,  or  Not 
Applicable.  If  a  primary  question  is  Not  Applicable,  its  corresponding  secondary  ques¬ 
tions  may  be  skipped. 

7.  The  last  five  lines  allow  for  the  program  phase  questions  to  be  tailored  to  a  specific  pro¬ 
gram.  If  there  are  applicable  questions,  type  them  into  the  blank  spaces  in  the  “Ques¬ 
tion”  column.  See  Fleishman-Mayer,  Arena,  and  McMahon,  2013,  for  a  discussion  of 
designing  and  properly  framing  questions  for  the  tool. 

a.  Organize  questions  such  that  secondary  questions  fall  directly  below  the  associated 
primary  question. 
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Figure  2.1 

Overview  Tab  of  the  Assessor  Tool 


Program  Name:  Example  Program  1 

System  Engineering  "V"  and  Defense  Acquisition  System 

Mission  needs  identification  I  n  p  A 

♦  IpCA  FRP 

Fielding 


MDD - 

Requirements  definition 

MS  A - — \  ,  V 

Conceptual  design 
Preliminary  design 


MS  B 

Detailed  design 


Production 

MS  C 


Testing 


IRR  |  Results  | 

€,  Integration 


Implementation 


Concept 

development 


Material 

solution 

analysis 


£ 


Technology 

velopment 


Engineering  and 
manufacturing 
development 


Production 

and 

deployment 


Operations 

and 

support 


NOTE:  All  abbreviations  can  be  found  in  the  Abbreviations  list. 
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b.  In  the  column  labeled  ID,  update  the  numbers  to  reflect  the  addition  of  second¬ 
ary  questions.  That  is,  ID  numbers  for  primary  questions  are  given  whole  num¬ 
bers,  while  secondary  questions  associated  with  the  primary  include  the  same  whole 
number  with  an  additional  decimal,  ascending  in  increments  of  0.1  (e.g.,  a  pri¬ 
mary  question  with  the  ID  9  will  have  secondary  questions  with  IDs  9.1,  9.2,  etc.). 
Note  that  skipping  this  step  will  cause  there  to  be  an  error  in  the  overall  relative  risk 
calculation. 

c.  In  the  relevant  artifacts  and  domain  area  columns,  fill  in  as  appropriate. 

d.  Next,  choose  the  level  of  importance  on  a  scale  from  1  (Little  Importance)  to  5 
(Extremely  Important)  for  each  of  the  custom  primary  questions.  Note  that  the 
importance  of  custom  secondary  questions  will  be  the  same  as  their  associated  pri¬ 
mary  as  long  as  all  steps  included  in  step  5  have  been  followed.  Thus,  there  is  no 
reason  to  assign  an  importance  level  to  secondary  questions. 

e.  For  all  custom  primary  and  secondary  questions,  choose  its  appropriate  “Assess¬ 
ment”:  Addressed,  Partially  Addressed,  Not  Addressed,  or  Not  Applicable. 


Table  2.1 

Sample  ASR  Assessor  Data  Entry  Phase 


Program  Name:  Example  Program  1 


Assessor  Name:  Joe  Smith 


Date:  6/13/2012 


ID  Question 


l.i 


2 


3 


3.1 


4.1 


4  2 


4.3 


4.4 


43 


7 


6 


9 

10 
11 
12 
13 


Does  the  CONOPs  identify  the  *eationsh:ps.  dependences  anddes  -ed  interfates 
envisioned  between  new  or  upgraded  systems  and  other  enisling  or  planned 
systems? _ 

Does  the  opera?  onal  views  (OV-l|  frame  the  operator  concept  (  what 
happens,  who  does  what  in  what  crder,  to  accomplish  what  goal)  and 
_ highlight  interactons  to  the  cnv  'onment  and  other  extc-nal  systems? 

Does  the  Initial  Capabilities  Document  explain  how  the  rec  Jirec  capabi  ties  a*e 
dependent  upon  interface  with  other  systems?  Does  it  also  define  nteroperability 
requirements  of  the  capabilities  in  terms  of  h  gh-ieve  Operational  Vcw(OV-l)? 

Are  the  system  ana/or  FoS/SaS  reiabilty.  maintainaoilgy,  availability  performance 
parameters  tdenfcfied? 


Kji  a  forma  DMSMS  program  been  estab  sned? 


Mas  a  draft  Systems  Er^neenr*  Plan  been  Oeveoped? 

Has  the  Electrpmagnctic  Spectrum  supportablity  assessment  fa:to*s  dcc' 
competed  and  subnr  tted  for  sacct-jm  suppprtabv  approval? 

Does  the  integrated  architecture  adhere  to  the  DoD  ret-centnc  strategies? 
ts  tie  system's  architecture  explicitly  documented  to  the  same  level  as  the 
systems  requirements?  Does  the  architecture  documentaton  aescnbe  the 
rationale  for  partition  ng  functionality  and  for  placing  key  architectural 
_ attributes  wrtnin  or  across  a  suaset  of  architectura  paunaa-es? _ 

Have  MSI  ssues  been  integrated  nto  the  systems  acQjisitipn 
documentation? 

Does  the  program  have  a  SoS  engineering  IPT? 


Have  software  testing  requirements  been  identif  ed? 


Have  the  requirements  for  an  irgeg'ated  test  facility  been  identified? 

Dd  the  AoA  performance  assessment  adequately  evaluate  integration 
issucs/FpS/SoS? _ 

Have  the  requirements  fpr  integration  test  activities  been  identfied  and  included 
mtne  cost  estimate? 


Technical  Relevant  Domain 

Review  Artifacts  Area 


Importance 


Assessment 


Score 


Weight 


Relative 

Risk 


Relative 
Risk  (un¬ 
weighted) 


CONOPs,  OV-1.  TDS 


Oesigp  and 
Engineering 


4  -  Very  Important  *  )  Partially  Addressed 


1  OS 


Design  and 
Engineerng 


Not  Addressed 


mitoi  capabilities 
document,  OV-1 


Design  and 
Engineer  ng 


(  2  *A  little enportant  )  )  Not  Appkcabe 


SEP,  initial  capaoilites 
document,  CONOPs, 
OV-1 


Logistics  and 
Mantainab:  ty 


I  3  -  Somewhat  import^)  Addressed 


logistics  and 
Mantainab  tv 


Partially  Addressed  J  1  0.S 


Program 

Management 


4  -  Very  Important  J  )  Partially  Addressed  J  ^  os 


Design  and 
Engmce-ng 


T) 


Net  Cent*  c  Data 
Strategy 


Design  and 
Engmce-  ng 


Addressed 


T) 


Design  and 
Engineerng 


Not  Addressed 


Program 

Management 


Test  and 
Evaluation 


[  5  -  Extremely  Important] 


ParUwy  Addressed  * 


Test  and 
Evaluaton 


-  Very  Important  *  } 


Design  and 
Eng  nee-ng 


[  3  -  Somewhat  imp  "-’♦‘I 


ParUswy  Addressed  $ 


Test  and 
Eval  uat  on 


[  3  -  Somewhat  imp 


Not  Addressed 


I  1  -  Very  little  ImporuQc^ 


TL 


-  Very  little  Import*^ 


TL 


[  1  -  Very  little  Important} 


Tl 


|  1  -  Very  little  ImpcirtaQc} 


Total  ASR  Relative  Risk 


0.04 


0.07 


0.00 


0.00 


0.04 


0.04 


0.07 


0.00 


0.00 


0.07 


0.00 


0.04 


0.00 


0.04 


0.07 


0.00 

0.00 

0.00 

0.00 

0.00 


0.46 


i~n 


NOTE:  All  abbreviations  can  be  found  in  the  Abbreviations  list. 
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8.  Scroll  down  on  the  page  to  find  the  global  questions  (highlighted  in  purple).  See 
Table  2.2.  For  each  global  question,  follow  the  same  instructions  in  steps  6  and  7  above. 
Note  that  “Assessment”  choices  may  be  somewhat  different  than  those  for  the  program 
phase  questions  (e.g.,  Yes,  Somewhat,  No,  Not  Applicable;  High,  Medium,  Low,  Not 
Applicable). 

9.  Scroll  to  the  very  bottom  of  the  program  phase  tab  to  reveal  a  button  labeled  “Results.” 
Click  the  “Results”  button  or  choose  the  results  tab  in  Excel  associated  with  the  appro¬ 
priate  program  phase. 

10.  Review  the  results  report  (e.g.,  Figure  2.2).  See  Fleishman-Mayer,  Arena,  and  McMa¬ 
hon,  2013,  for  a  further  discussion  of  the  results  report.  The  buttons  at  the  top  of  the 
page  allow  for  the  report  to  be  printed  and  will  direct  the  user  back  to  the  home  page 
or  back  to  the  program  phase  questions. 


Adapting  the  Assessor  Tool  for  Other  Programs  or  Information-Based  Risk 
Assessments1 

This  section  contains  step-by-step  instructions  for  a  user  of  the  Assessor  Tool  template.  The 
instructions  will  lead  the  user  through  an  exercise  to  create  a  tailored  Assessor  Tool,  such  as 
the  example  integration  risk  Assessor  Tool  described  in  Fleishman-Mayer,  Arena,  and  McMa¬ 
hon,  2013.  A  user  can  use  these  instructions  to  create  Excel  tabs  to  hold  questions  about,  and 
results  for,  multiple  program  phases.  For  instructions  of  how  to  assess  the  program  associated 
with  this  adapted  tool,  a  user  may  loosely  follow  the  instructions  presented  previously  for  the 
integration  risk  application  of  the  Assessor  Tool. 

1.  Open  the  Assessor  Tool  template  file  (see  Table  2.3). 

2.  If  the  program  of  interest  has  more  than  one  program  phase,  make  copies  of  the  existing 
questions  and  results  tab. 

a.  Rename  the  tabs  to  reflect  program  phases.  For  example,  for  the  ASR  program 
phase,  the  questions  tab  could  be  named  “ASR,”  and  the  results  tab  could  be  named 
“ASR  Results.” 

b.  Go  to  each  results  tab  and  change  all  formula  references  to  the  corresponding  ques¬ 
tions  sheet. 

•  One  way  to  do  this: 

-  Show  all  formulas  in  the  tab  using  the  “show  formulas”  option/command  in 
Excel.2 

-  Using  “Find  and  Replace  All,”  change  the  tab  reference  “Questions  Template” 
to  the  name  of  the  appropriate  questions  tab  (e.g.,  ASR). 

-  Hide  all  formulas. 


1  Note  that  these  instructions  assume  moderate-level  Excel  skills. 

2  To  show  formulas  in  Excel  2012,  go  to  the  File  menu  and  select  Options.  It  will  bring  up  the  Excel  Options  dialog. 
From  the  left  sidebar,  click  Advanced,  and  from  the  right  pane  scroll  down  to  find  Display  options  for  this  worksheet  group. 
Under  this  group,  enable  the  “Show  formulas  in  cells  instead  of  their  calculated  result”  option.  Click  OK  to  continue.  To 
show  formulas  in  Excel  2007  for  the  Mac,  go  to  the  Excel  ribbon  titled  Formulas,  and  under  the  Function  heading,  select 
the  Show  label. 


Table  2.2 

Sample  ASR  Assessor  Data  Entry  Phase,  Global  Questions 


Program  Name:  Example  Program  1 


Assessor  Name:  Joe  Smith 


Date:  6/13/2012 


ID  Question 


Relative 

Technical  Relevant  Domain  ,  _  _  ,  ...  .  . ,  Relative  , 

„  ,  .  Importance  Assessment  Score  Weight  ,  Risk  (un- 

Review  Artifacts  Area  Risk  .  '  .. 

weighted) 


G1 

Has  a  schedule  breach  been  reported  in  the  program  in  0  AMIR/DASH  BOARD 
reporting? 

ASR 

[  3  -  Somewhat  important] 

:) 

1 

3 

0.0B 

0.07 

Was  the  schedule  breach  due  to  major  system  development,  deliveries  or 
production  that  increase  integration  risk? 

GL1 

ASR 

Somewhat 

: ) 

0.5 

3 

0.04 

0.04 

GL2 

Has  the  IMS  changed  due  to  the  breach  such  that  integration  risk  has 
increased  due  to  reduced  durations  for  installation  and  testing? 

ASR 

0 

1 

3 

o.os 

0.07 

G2 

Have  f  jnding  changes  increased  integration  risk  due  to  inadequate  f  jnds  'or 
testing  or  resulted  in  delays  for  technology  insertion? 

ASR 

[  2  -  A  little  important  J  ] 

No 

0 

0 

2 

0.00 

0.00 

G3 

Has  the  program  been  identi'ied  for  a  Nunn-McCurdy  breach? 

ASR 

4  -  Very  Important  j] 

No 

: 

o.s 

4 

0.06 

0.04 

For  any  changes  as  a  result  o‘  N-M  recertification,  have  there  been  any 
program  technical  risks  or  integration  risks  that  have  not  been  assessed? 

G3I 

ASR 

>> 

:) 

1 

4 

0.11 

0.07 

Are  there  any  technology  risk  issues  identified  in  the  most  recent  ADM  that  have 
not  been  addressed  by  the  program? 

G4 

ASR 

[  3  -  Somewhat  important] 

Somewhat 

0 

0.5 

3 

0.04 

0.04 

G5 

What  is  the  DOT&E  risk  assessment  for  the  program  (M,  M,  L|? 

ASR 

[  2  -  A  Irttle  important  J  ] 

(  High 

0 

1 

2 

0.06 

0.07 

G5.1 

Has  any  of  the  TEMP  not  been  concurred  and  funded? 

ASR 

:) 

1 

2 

0.06 

0.07 

Has  a  major  sub-system  or  technology  vendor  failure  occurred  or  vendor  been 
disqualified  or  changed? 

G6 

ASR 

1  -  Very  Little  Importa^c^ 

Not  Applicable 

:) 

0 

0 

0.00 

0.00 

G6.1 

What  is  the  associated  integration  risk  effect  on  the  program  (H.M,L)? 

ASR 

[  Medium 

:) 

0 

0 

0.00 

0.00 

Have  all  required  certifications  or  the  planning  to  achieve  required  certifications  at 
the  appropriate  time  been  addressed? 

G7 

ASR 

[  2  -A  little  important  $) 

[  Some 

:) 

0.5 

2 

0.03 

0.04 

G7.1 

Information  Assurance  (eg.  DIACAP  and  NSA  Cryptographic  Certfication)  ? 

ASR 

l" 

:) 

0 

2 

0.00 

0.00 

G7.2 

Interoperability  (eg.  Net  Ready  KPP,  Joint  Interoperability  Test  Certificate)? 

ASR 

[  Some 

:) 

O.S 

2 

0.03 

0.04 

G7.3 

Spectrum  Management  |  e.g.  EMI  /  EMC  Cert,  Spectrum  Cert.)? 

ASR 

“ 

0 

0 

2 

0.00 

0.00 

G7.4 

Safety  (eg.  Airworthiness,  SU85AFE.  PESHE)? 

ASR 

None 

0 

1 

2 

0.06 

0.07 

G8 

ASR 

[  1  -  Very  Little  lmporta£^ 

> 

:) 

1 

0 

0.00 

0.00 

G9 

ASR 

1  -  Very  Little  Iniportaflt^ 

:) 

1 

0 

0.00 

0.00 

G10 

ASR 

1  -  Very  Little  Impurtafc^ 

( 

:) 

1 

0 

0.00 

0.00 

Gil 

ASR 

1  -  Very  Little  Inipurta#:^ 

(  Yes 

T) 

1 

0 

0.00 

0.00 

Total  Global  Relative  Risk 

0.64 

0.61 

Back  to  Home  Page 


NOTE:  All  abbreviations  can  be  found  in  the  Abbreviations  list. 
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Figure  2.2 

Results  Page  for  Sample  Run 


RAND  TL113-2.2 


c.  Change  data  references  and  labels  in  the  graph. 

•  To  find  data  references:  Right  click  with  the  pointer  over  the  graph,  choose  “select 
data.” 

•  To  change  the  labels,  it  may  be  necessary  to  download  an  Excel  add-in.3 
3.  For  each  questions  tab: 

a.  After  organizing  questions  such  that  secondary  questions  fall  directly  below  the 
associated  primary  question,  type  phase-specific  questions  in  the  appropriate  cells 
(into  the  blank  spaces  in  the  “Question”  column).  See  Fleishman-Mayer,  Arena,  and 
McMahon,  2013,  for  a  discussion  of  designing  and  properly  framing  questions  for 
the  tool. 

b.  In  the  column  labeled  ID,  update  the  numbers  to  reflect  the  addition  of  second¬ 
ary  questions.  That  is,  ID  numbers  for  primary  questions  are  given  whole  num¬ 
bers,  while  secondary  questions  associated  with  the  primary  include  the  same  whole 


3  In  Microsoft  Excel,  there  is  no  built-in  command  that  automatically  attaches  text  labels  to  data  points  in  an  xy  (scatter) 
chart.  However,  you  can  create  a  Microsoft  Visual  Basic  for  Applications  macro  that  does  this.  There  is  an  xy  labeler  add-in 
that  does  not  require  writing  your  own  macro.  See  Application  Professionals,  not  dated. 


Table  2.3 

Assessor  Tool  Template  File 


Program 

Name: 


Program 

Assessor 

Example  Program 

6  Stage  1 

Stage: 

Name: 

Joe  Smith 

Date: 

ID 

Question 

Relevant 

Artifacts 

Domain  ,  . 

Importance  Assessment 

Area 

Score 

Weight 

Relative 

Risk 

Relative  Risk 
(un¬ 
weighted) 

STAGE-SPECIFIC  QUESTIONS 

1 

Replace  with  question  1 

1  -  Very  Little  Importance  )  Partially  Addressed 

OS 

1 

0.08 

0.13 

1.1 

Replace  with  question  1.1 

[  2  -  A  Mile  important  J  }  Partially  A:i  i  • 

o.s 

1 

0.08 

0.13 

2 

Replace  with  question  2 

3  -  Somewhat  tfnportant  i  '  Not  Addressed 

i  1 

1 

3 

o.so 

0.25 

a 

Replace  with  question  3 

1  -  Very  Little  Important  )  Addressed 

0 

0 

1 

0.00 

0.00 

s 

1  -  Very  Little  Importance  )  Not  Applicable 

:l 

0 

0 

0.00 

0.00 

6 

1  •  Very  Little  Importance  )  Not  Applicable 

:) 

0 

0 

0.00 

0.00 

7 

3  -  Somewhat  important  C  Not  Applicable 

i) 

0 

0 

0.00 

0.00 

8 

1  Very  Little  Imports -ica  Nut  Applicable 

:) 

0 

0 

0.00 

0.00 

9 

1  -  Very  Little  Importance  '  Not  Applicable 

:) 

0 

0 

0.00 

0.00 

10 

|  1  -Very  Little  Importance  ]  .  Not  Applicable 

i) 

0 

0 

0.00 

0.00 

11 

[  2*  A  Mile  important  ■  X  )  [  Not  Applicable 

H 

0 

0 

0.00 

0.00 

12 

2  -  A  little  important  !  [Not  Applicable 

;) 

0 

0 

0.00 

0.00 

ia 

(  1- Very  Little  Importance  1  Not  Applicable 

i) 

0 

0 

0.00 

0.00 

14 

1  -  Very  Little  Importance  ^  Not  Applicable 

:) 

0 

0 

0.00 

0.00 

RAND  TL113-T2.3 
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number  with  an  additional  decimal,  ascending  in  increments  of  0.1  (e.g.,  a  pri¬ 
mary  question  with  the  ID  9  will  have  secondary  questions  with  IDs  9.1,  9.2,  etc.). 
Note  that  skipping  this  step  will  cause  there  to  be  an  error  in  the  overall  relative  risk 
calculation. 

c.  In  the  relevant  artifacts  and  domain  area  columns,  fill  in  as  appropriate. 

d.  Note  that  the  template  allows  for  the  inclusion  of  50  phase-specific  questions  and 
30  global  questions  per  program  phase. 


CHAPTER  THREE 

Conclusion 


This  document  presented  the  User  Manual  for  the  Assessor  Tool,  which  can  be  used  to  facili¬ 
tate  an  OSD-level  information-based  risk  assessment  for  acquisition  or  other  major  programs. 
A  complementary  report  describing  the  methodology  behind  the  tool  and  its  applications  is 
available  as  RR-262-OSD,  A  Risk  Assessment  Methodology  and  Excel  Tool  for  Acquisition  Pro¬ 
grams  (Fleishman-Mayer,  Arena,  and  McMahon,  2013).  The  tool  includes  a  generalizable  form 
of  the  Assessor  Tool  as  well  as  the  integration  risk  Assessor  Tool  provided  as  an  example  appli¬ 
cation.  The  reproducible  and  documented  tool  for  integration  risk  assessment  may  be  consid¬ 
ered  for  program  office  reporting  to  meet  WSARA  compliance  as  well  as  for  other  acquisition 
reviews,  such  as  the  OSD  Defense  Acquisition  Executive  Summary  and  Overarching  Inte¬ 
grated  Product  Team  reviews,  and  for  adaptation  into  other  program  assessment  tools,  such  as 
the  Probability  of  Program  Success  tool.  As  of  this  writing,  the  Assessor  Tool  has  not  yet  been 
validated  in  a  real-world  setting.  Potential  future  work  could  include  its  validation. 
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